SPATIAL ASSET MANAGEMENT SYSTEM AND METHOD 



CROSS REFERENCE TO RELATED APPLICATION 
This application is a divisional application of Application 
No. 08/714,583, filed on September 16, 1996, which is hereby 
incorporated herein by reference. 

FIELD OF THE INVENTION 
This invention relates to methods for combining Global 
Positioning System (^'GPS") , Speech Recognition, Radio Frequency 
(''RF") , and Geographic Information System ("'GIS") to perform mobile 
field data collection and automatic population of a GIS database 
with fully attributed and correlated observation data. The system 
relates particularly to a field data capture system and automatic 
GIS database population tool for a user to build GIS layers and 
fully exploit the data in the GIS. 

BACKGROUND OF THE INVENTION 
Organizations responsible for the maintenance and inventory of 
assets are turning to GIS as the tool of choice to manage and 
display these assets. Over eighty percent of the cost of a GIS is 
capturing and placing accurate, fully attributed data into the GIS. 
These costs have prohibited many users from either implementing or 
fully exploiting the GIS. 



A number of different methods have been developed for 
capturing data in the field. Many users use the data collection 
method of traveling an inspection route, visually identifying the 
location, and hand writing a description onto a form or a paper 
entry. Once the inspector returns to the central data repository 
the entries so collected are manually entered into a database with 
questionable accuracy and time consuming labor. The user must 
build the correlation and association logic into the database to 
create a useful tool. Back end applications must also be created 
so that the information is useful to the user. More sophisticated 
methods include GPS with push button data collection or pen 
computer data entry units which allow predefined buttons and menus 
to be used for field data collection. The data can be 
electronically downloaded into a database, but a user must still 
build the correlation and association logic. The information 
downloaded is limited to point information with limited attribute 
information. 

Audio based data entry systems have been developed but are 
limited to the recording of street point information sequenced with 
a manually recorded location input. The user is then required to 
manually convert, transfer, and combine the location data with the 
audio data. There is no processing of the audio data and manual 
transcription, and tagging of the entries with location data must 
be manually performed by the user. Only location data where a 
observation has been recorded is stored, and all other location 
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information is ignored. Other speech recognition systems require 
the user to prerecord their speech to replace keyboard entries. 
None of the described systems provide the automatic population of 
the GIS with fully attributed and correlated data generated from 
5 speech recognition. 

As users of spatial data incorporate GIS and GPS based 
technology, the need for a flexible, true end to end system that 
collects field data, populates a GIS, tracks field assets, and 
provides tools to exploit the data will increase. 

^10 

tfl SUMMARY OF THE INVENTION 

0'^ It is an object of the present invention to provide a method 

and system for a speech recognition based field data capture 
^ system, asset tracking, and automatic GIS database population tool 
CjiS for a user to build GIS layers, to track assets, and to fully 
m exploit the data in the GIS. 

H It is an object of the present invention to combine GPS, 

Speech Recognition, and GIS, and to provide field data collection, 
automatic GIS database population, and exploitation of the GIS 
20 data . 

It is an object of the present invention to provide the real 
time tracking of assets in the field through the combination of GPS 
and RF communications. 

In furtherance of these objects, a field mobile unit capable 
25 of continuously capturing feature observations from predefined 
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grammar and free speech, as well as GPS based location information 
time-stamped and automatically stored on the units onboard memory, 
is created. Location information is automatically corrected in the 
field using Differential Global Positioning Service (''DGPS") and RF 
wireless data transmission. The location information is 

automatically combined with observation information to provide a 
continuous record of locations and observations. 

The preferred mobile field unit device is mounted in a vehicle 
or backpack. The audio headset microphone provides the means for 
initiating a speech-based description of user observations. The 
mobile unit computer provides the onboard data storage of speech 
observations and the GPS time-stamped location signal. The unit 
provides the ability to electronically transfer field data. The 
unit provides an audio feedback to the user to optimize speech 
entry start and stop, as well as notify the user of loss of GPS 
signal. The grammar structure provides self editing tools as well 
as a comprehensive description of field observations. 

In the preferred form of the invention the location and 
observation information is transferred electronically to the 
central data repository or via RF wireless media. The audio data 
process automatically converts the audio data collected in the 
field using the semantic information in the reference grammar and 
creates data records representing the information content of the 
user's verbal observations. The user can validate and correct 
observation statements. Interactive tools allow the user to review 
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all speech entries and correct them as required. The results are 
user validated and grammatically valid. 

The preferred form of the invention automatically merges the 
corrected location data and the recognized text data and precisely 
synchronizes the verbal data to a location, as well as identifying 
any continuous span of tracks covered by an observation. The data 
is then automatically entered into the GIS database and correlated 
to linear networks and point observations within the central data 
repository. 

The preferred form of the invention provides predefined or 
customer configurable tools to exploit the data in the central data 
repository. Work orders, custom reports, and data query scripts 
are created using these tools. 

The vehicle location information is derived from GPS which 
provides a time-stamp from which absolute location coordinates may 
be determined through interpolation of recorded GPS data points. 

Methods and apparatus which incorporate the features described 
above and which are effective to function as described above 
comprise specific objects of the present invention. 

Other and further objects of the present invention will be 
apparent from the following description and claims and are 
illustrated in the accompanying drawings, which by way of 
illustration, show preferred embodiments of the present invention 
and the principles thereof and what are now considered to be the 
best modes contemplated for applying these principles. 



60063207_2DOC 



-5- 



BRIEF DESCRIPTION OF THE DRAWING VIEWS 
Figure 1 is a diagrammatic view of a spatial asset management 
system constructed in accordance with one embodiment of the present 
invention. Figure 1 shows the processes, the data elements used in 

5 the processing, and the user interaction with the system. Figure 1 
is a high level overview of the system. 

Figure 2 is a diagrammatic view showing the details of the 1.0 
Data Conversion process of Figure 1. Figure 2 shows the 1.0 Data 
Conversion processing in conjunction with the collected data 

10 elements and the reference data elements and the user interaction. 
Figure 2 shows both the Audio Data l.A and GPS Data l.B going 
through their appropriate processing paths and being merged into an 
Observation l.G. Figure 2 also shows, in the component labeled 
Track l.F, the historical representation of where the field 

15 operator had been and when the field operator had been there. The 
Observation l.G and the Track l.F are two key outputs of the 1.0 
Data Conversion process shown in Figure 2. Semantic analysis is 
performed in the 1.6 Interpret Text process and by use of the 
Reference Observation Semantics l.E to create the Observation l.G. 

20 Figure 3 is a diagrammatic view showing details of the in the 

2.0 Data Correlation process of Figure 1. Figure 3 shows the two 
main data inputs (the Track l.F and the Observation l.G) coming 
from the 1.0 Data Conversion process shown in Figure 2. Figure 3 
shows that Track l.F is first correlated to the Reference Network 

25 l.K. Figure 3 also shows that the input information Track l.F and 
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Observation l.G are correlated to the Reference Network l.K and to 
the appropriate other layers of the GIS creating a Tour l.L object. 
The Tour l.L object comprises: who collected the data; what data 
was collected; where the field operator was; what the field 
5 operator was doing; when the field operator was collecting the 
data; and the correlation results. 

Figure 4 is a diagrammatic view showing the 3.0 Repository 
Update process as updated with the Tour l.L results. Figure 4 also 
shows, the 3.3 Define Repository process and the 3.5 Configure Tour 
.=ao process, the definition of the repository structure. 
'M Figure 5 is a pictorial view, in plan, showing an example of 

data collection in the field. Figure 5 shows a vehicle traveling 
Si north on Elm Street. Figure 5 shows the position of the vehicle by 
r' its GPS points and shows two observation events indicated by the 
11115 numerals 1 and 2. The data input from the observation events is 
y voice data, indicated by the quotations in Figure 5, 
Jl' Figure 6 shows the processing sequence for data conversion for 

the two specific observation events identified in Figure 5. 
Figure 6 also shows the semantic analysis of associating 
20 observation event 2 to observation event 1. The results of the 
semantic analyses are indicated by the inclined block arrow in the 
lower part of Figure 6. 

Figure 7 is a diagrammatic view illustrating the four primary 
types of data maintained within the Repository l.M of the system 
25 shown in Figure 1. In Figure 7 the arrows indicate the data 
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structure relationships. As illustrated in Figure 1, Assets can 
always be associated with other Assets, Condition must be 
associated with an Asset, Defect must be associated with an Asset, 
and Repair can be associated only with a Defect. Figure 7 also 
5 shows the structure for each of the primary data types. The 
processing information portion of the structure of each primary 
observation type is embodied in the association (indicated by the 
arrows) , the Spatial Type information, and the Storage Layer and 
Associated Layers information. Each of the primary observation 
10 types also have Location and Attributes in its structure. 

Figure 8 requires too much illustration area to be capable of 
being shown on one sheet of drawing and is therefore composed of 
Figure 8A (on one sheet of drawings) and Figure 8B (on the 
succeeding sheet of drawings) . Figure 8 is an example grammar of 
15 the type used in Figures 5 and 6 but for a specific asphalt 
distress observation type. Each of the boxes shown in Figure 8 
represent different sentence types. The two observation events 
illustrated in Figure 5 correspond to the respective top box and 
bottom box in Figure 8. The semantic information identifying that 
20 the second sentence is a modifier of the first sentence is 
illustrated by the two dashed lines in Figure 8 — the first dashed 
line going from ''Tag: blob" up to the term ''blob" and the second 
dashed line going from "Tag: area" up to the term "area" in the 
Observation Template. The observation statements in Figure 5 
25 correspond to the Recognized Text 2. A in Figure 2, and the 
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Reference Observation Semantics l.E of Figure 2 correspond to the 
information contained in the Asphalt Project Grammar of Figure 8. 

Figure 9 is an illustration of the 2.0 Data Correlation 
process using the example illustrated in Figure 5 and continuing 

5 the example shown in Figure 6. Figure 5 shows data collection. 
Figure 6 shows data conversion. Figure 9 shows data correlation. 
Figure 9 shows how an observation in track data is correlated to an 
asset (note the results of the correlation show that the Defect is 
correlated to the street segment on Elm Street between First Street 

10 and Second Street) . Figure 9 also illustrates the process of 
moving data into the appropriate GIS layers in the spatial asset 
management system of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 
15 Figure 1 presents an overview of a preferred form of the 

spatial asset management system and method. Subsequent Figures 2-4 
expand each major process shown in Figure 1. For example, the 
process 1.0 Data Conversion (the top circle in Figure 1) is 
expanded into a more detailed flow chart in Figure 2. 
20 The spatial asset management system and method described 

herein is a hardware independent system solution for managing 
assets with a strong geospatial component. The preferred form of 
the system is implemented in a commercial off-the-shelf laptop or 
pen-based computer for the mobile system component and a high 
25 performance PC for the processing workstation home base computer. 



60063207 2DOC 



-9- 



The three data stores Audio Data l.A, GPS Data l.B, and Sensor 
Data l.C shown in Figure 1 are generated in the mobile system 
laptop computer. All subsequent processes and data stores are 
maintained in the home base computer or workstation. 

The system provides for a seamless, fully automatic capture, 
translation, GIS import, and analysis/processing of asset 
information as represented by the Audio Data l.A, GPS Data l.B, and 
Sensor Data l.C stores developed during collection. The mobile 
unit computer may be hand carried (e.g., backpack) or mounted on a 
moving vehicle (e.g., car, truck, bicycle). 

Figure 5 illustrates the collection of data whereby the user 
can drive, or walk along, an inspection route and can comment on 
observed defects, assets, asset condition or other observations. 
Also shown in Figure 5 are the GPS points that are collected by the 
system. 

Figure 6 shows how the observations in Figure 5 are processed 
functionally by the system to become data items that are correlated 
against existing asset information and analyzed for corrective 
action by operations personnel. 

The mobile unit computer is configured with a commercial GPS 
receiver (or other location receiver device) , a standard commercial 
sound board, and standard I/O devices (e.g., printer, disk drive, 
RS-232 ports) along with a battery or external power source. Other 
sensor inputs include such sensors as digital cameras, laser 
ranging devices, and others. For example, digital camera sensor 
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input allows for photos to be included of city code violations. In 
this case the digital photo image is automatically tagged and 
tracked by the system so that photo evidence is included directly 
in the violation report sent to the offender. 
5 Voice observations are automatically correlated with the 

sensor inputs to be incorporated as an associated data record. The 
mobile unit computer captures and time-stamps all data store 
records- Each data store is independent and no other synchronized 
signal or input is required other than standard precision time. 
10 The Audio Data l.A store contains all speech audio data detected by 
the mobile system sound card. The GPS Data l.B store includes 
location derived information containing latitude, longitude, and 
altitude of the mobile unit on a continuous basis once the unit is 
initialized. The Sensor Data l.C store contains any external 
15 sensor records, such as switch on/off states, analog values, 
digital photos, laser ranging data, etc. 

As will be described in more detail with reference to 
Figure 2, the 1.0 Data Conversion process means receives the mobile 
unit data from Audio Data l.A, GPS Data l.B, and Sensor Data l.C 
20 data stores described above. The 1.0 Data Conversion process 
operates on these inputs in conjunction with reference data 
(Reference Grammar l.D, Reference Observation Semantics l.E, and 
Reference DGPS Data l.H) to produce Track l.F objects and 
Observation l.G objects data stores. The functions supported by 
25 the 1.0 Data Conversion process are: (1) automatic interpretation 
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of audio data spoken words using a reference dictionary contained 
within the Reference Grammar l.D data store, (2) automatic 
detection of word level interpretation error conditions, 
(3) automatic interpretation of phrases using pre-defined meaning 
and phrase syntax contained within the Reference Observation 
Semantics l.E data stores, (4) automatic detection of semantic 
error conditions, (5) optional correction of GPS location data 
using Reference DGPS Data l.H, (6) automatic generation of time 
based location Track l.F data objects in internal system format, 
(7) automatic generation of time-based Observation l.G data objects 
and internal system format and (8) operator use of interactive 
displays to perform Quality Assurance (QA) functions against either 
Audio Data l.A or Sensor Data l.G stores. 

The net result of the 1.0 Data Conversion process is a data 
store of error corrected track information which is an automated 
time-sequenced track of the mobile unit's historical travel path 
with precise latitude, longitude and altitude for a given "'Tour" 
(note that Tours are actually generated by the 2.0 Data Correlation 
process) . 

Another result of 1.0 Data Conversion process is a time- 
sequenced and operator quality assurance checked set of observation 
objects, which represent either 'Miscrete" observations (e.g., 
^^tree, foliage damage," ''stop sign, class 1 damage," ''pothole, 
high, right"), "linear" observations (e.g., "start curb and gutter 
run," "end curb and gutter run," "start road width 32," "end 
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road"), or "polygon" definitions which is a unique form of track 
data store. These Track l.F and Observation l.G data stores are 
available to the 2.0 Data Correlation process. 

Figure 6 illustrates the buildup of these data types. The 
system organizes data into a logical contiguous set of collected 
data that may last from a few minutes to several hours. A street 
inspection tour, for example, would typically consist of the 
collection of street distress data for several hours before 
concluding the collection and submitting the collected data to the 
home base workstation for processing. 

The "discrete" observations captured include any and all 
assets which are best categorized as an item or set of items at 
discrete locations. Examples of types of objects collected are 
signage, lights, street distresses, concrete distresses, park 
benches, tree damage, utility cuts, utility access covers, fire 
plugs, incidences of code violations (e.g., weeds, illegal cars 
parked, damaged fence, etc.), curb damage, sidewalk distresses, and 
other items of the like. Usually discrete object observations are 
accompanied by a status, state, or condition which are related to 
the object and position, a size, or other descriptive term that may 
help identify or qualify the observation. The phrase "pothole, 
medium, right," would be translated by the 1.0 Data Conversion 
process to mean: 

"pothole" = pothole type of road distress; 

"medium" = distress severity medium; 
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'"right" = the right lane (assuming more than one lane in the 
current direction of travel) . 

Similarly "'linear'' observations are used for assets or objects 
that are running or continuous in nature for some significant 
5 length. Examples are roads, sidewalks, curbs, gutters, fences, 
paint stripping, property frontage, and others of the like. Linear 
objects are usually accompanied by state or condition, plus an 
indication that the running asset starts or stops at some position. 
An example might be when an inspector is monitoring the 
10 condition of road centerline paint conditions. A phrase may be 
'"start road centerline paint condition 3" which would mean that the 
inspector is reporting the beginning of a class 3 (e.g., badly 
worn) status of road stripping condition. This condition may last 
for several miles. When the condition changes the inspection would 
15 terminate the running asset condition with a phrase such as "end 
road centerline condition 3." 

The system interprets and keeps track of all running asset 
states. In addition the inspector may continue commenting on any 
other objects or observations while the linear conditions are being 
20 tracked. That is to say that the inspection can start a running 
asset observation (like the road paint stripping), then report on 
several defects (such as sign damage), and then terminate the 
running asset conditions. The system automatically keeps track of 
all such interleaved conditions. Logic errors are automatically 
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detected and identified to the operator during the Quality 
Assurance processing with the 1.0 Data Conversion process. 

Another observation data type is "polygonal." Polygonal data 
is usually associated with defining areas or boundaries. Using a 
5 backpack mounted system, a parks inspector might, for example, walk 
and define the boundaries of an area of a park, perform a tree or 
endangered species inventory or forest damage by some infestation. 
The results would be a polygon that describes the area where the 
observations are located. 
10 As described in more detail below, the 2.0 Data Correlation 

process means operates on the Track l.F and Observation l.G data 
stores which are output by the 1.0 Data Conversion process means to 
perform correlation against a variety of reference data. The 2.0 
Data Correlation process organizes and associates Track l.F data 
15 stores with Observation l.G data stores which are output to produce 
logical "tours," which are sets of data (collected by the user) 
such as those discussed earlier. 

The 2.0 Data Correlation process automatically routes data 
items to the proper layer of the CIS database for further 
20 processing. For example, signage would be associated with a 
specific layer of CIS whereas street distresses would be associated 
with a separate layer. The 2.0 Data Correlation process uses the 
Reference Asset l.J data store to correlate the collected discrete 
asset observation tour data to the existing database of objects 
25 (e.g., signs, park benches, etc.) of the same category or class. 
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The system automatically detects inconsistencies between the 
collected and reference asset data and brings problems to the 
attention of the field operator. These inconsistencies can be 
corrected or edited using Quality Assurance tools provided. 
5 Ultimately the reference asset database is updated for future 
reference , 

Similarly, observation tour data which represents discrete 
defects, (e.g., road potholes, fence damage, curb upheaval, etc.) 
are correlated and compared against the Reference Defect l.I data 

10 store and are quality assured for consistency and logical error 
state by the 2.0 Data Correlation process. The 2.0 Data 
Correlation process also performs the same type of functions for 
linear observations tour data, such as curbing and sidewalk 
networks, using the Reference Network l.K data store. A set of 

15 Edit and Quality Assurance tools are provided to support the 
correlation processing of network type data. 

Reference Network l.K data stores include simple tour location 
Track l.F data as well (which allows the system to capture and 
compare location track data independent of collected discrete, or 

20 linear objects) . This enables the system to identify which 
inspectors have inspected which streets and when. It also allows a 
broad range of tour analysis functions to be accomplished, such as, 
which areas have streets that have not been inspected for the last 
three months. 
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The general functionality supported by the 2.0 Data 
Correlation process are (1) automatic association of collected data 
to proper GIS layers, (2) automatic detection of inconsistencies 
between collected observations and reference data, (3) correction 
5 of conflicted data, (4) analysis of tour location track information 
such as routes traveled with temporal reference, (5) quality 
assurance of correlated data, and (6) the organization and 
association of Track l.F and Observation l.G into ''tours" which are 
correlated location, observation, and time data sets. 
10 The 3.0 Repository Update process means provide all of the 

tools to create, update, and generally manage the system reference 
databases. A primary input to this process is the Tour l.L data 
store which is generated by the 2.0 Data Correlation process. The 
3.0 Repository Update process provides the tools to create new 
15 assets and/or conditions the system will recognize by updating the 
Reference Grammar l.D data store and the Reference Observation 
Semantics l.E data store along with the appropriate Reference Asset 
l.J, Reference Defect l.I, or Reference Network l.K data stores. 
Using this function allows the user to add new types of defects 
20 {such as a new type of damage or new class of utility cut in the 
road), add new asset types, add new tour types (such as utility 
inspection tours) , and any other operational data elements needed. 

Data management tools include editing, data consistency 
checking, data integrity and version control, and backup tools. 
25 Operational data store elements are maintained in the Repository 
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l.M database. The Repository l.M data store is where the results 
of system processing are placed. 

Using a variety of GIS configured, third party, and Spatial 
Asset System tools, the field operator/user can gain access to the 
operational database for analysis and reporting purposes. The 
analysis and reporting tools include both ad-hoc and predefined 
analysis and reporting capabilities. They range from such 
capabilities as visual analysis and interrogation of GIS layers to 
specific reports on such elements as road defect history in a given 
neighborhood. 

The user can query and generate reports on any and all data 
contained within the Repository l.M data stores. Using these tools 
the user can ask such questions as: 

How many of a specific asset type is located within center 
boundaries? 

What are the specific work orders (time to accomplish, etc.) 

to repair specified road segments? 
Show the inspection routes covered by a specified inspector 

over a given period of time. 
Show all road signs that are severely damaged and what is an 

optimal route for repair. 
Figure 2 is a detailed diagrammatic view of the 1.0 Data 
Conversion process of Figure 1. From the field collection process 
the results of the operator' s verbal inputs are represented by the 
data store labeled Audio Data l.A. These are time-stamped digital 
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audio data segments corresponding to each verbal phrase spoken by 
the field operator. 

The data store identified by the label GPS Data l.B represents 
all of the GPS data collected in the field during the operator's 
trip. The Reference DGPS Data l.H store is the DGPS correction 
data collected during the operator's trip. 

The 1.1 Correct Location Bias process applies the correction 
data to the GPS data, if it was not corrected in the field using 
real-time DGPS. Note that in the preferred implementation the 
field GPS units can be used in either real-time DGPS mode or post- 
processing DGPS mode, depending upon the needs of the field 
operator. 

The results of the 1.1 Correct Location Bias process is DGPS 
corrected location data that is then stored in the Corrected 
Location 2.B data store. The corrected data is then processed, by 
1.2 Vectorize Location Data, to convert the individual point data, 
(typically collected at 1 second intervals, but any interval period 
is possible), into track data which is stored in Track l.F. The 
purpose of this processing is to compress the point data into a 
higher order representation of linear and arc based tracks. This 
compression greatly improves the performance of latter processing 
illustrated in Figure 3. 

The 1.3 Recognize Audio Data process automatically converts 
the Audio Data l.A collected in the field using the semantic 
information in Reference Grammar l.D, and creates intermediate data 
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records (Recognized Text 2. A) representing textually/linguistically 
the information content of the operator's verbal statements made in 
the field. Note that the field unit can record the audio data in 
either of two ways. First, it can recognize when voice is present 
and only record when the operator is speaking, which is the 
preferred approach. Or the field unit can record all data 
regardless of whether the operator is speaking. 

In the latter case, the 1.3 Recognized Audio Data process will 
break the continuous audio data into the individual spoken phrases 
using the same approach as the field unit would use, i.e., energy 
threshold of the audio data. The user then can validate and 
correct any problems with the results through the 1.4 Verify Speech 
Recognition process. With the interactive tools provided in this 
process the user can review all of the automatic recognition 
processing and fix any problems encountered. 

The Reference Grammar l.D information is used to maintain the 
integrity of the resulting fixes. The Track l.F information is 
used to provide visual location information to the operator on 
where they were at the time they made the verbal statement. The 
results from 1.4 Verify Speech Recognition processing are stored 
into Recognized Text 2. A. These results are both user validated 
and grammatically valid. 

The 1.5 Assign Location process automatically merges the Track 
l.F data and the Recognized Text 2. A data, precisely synchronizing 
the verbal data to the location data and identifying any contiguous 
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span of tracks covered by an observation. The resulting merged 
data is forwarded to the 1.6 Interpret Text process. This process 
uses the Reference Observation Semantic l.E information to merge 
the sequence of recognized text into actual Observation l.G. 

5 It should be noted that the system can take a non-contiguous 

set of verbal statements and combine them into a single 
observation. An example of this process is discussed latter, 
relative to Figure 8. 

The 1.6 Interpret Text process performs the semantic analysis 

10 on the sequence of recognized text to determine if it is complete 
and consistent. 

Figure 4 is the diagrammatic view of the repository 
maintenance functions. The user interacts with the system through 
these functions to define the data to be collected and merge the 
15 collected data into a central data Repository l.M. The user 
interacts with three functions to perform repository maintenance. 

The user, through a series of displays in 3,3 Define 
Repository process, defines the data to be collected and the 
grammars with semantics used to process the collected field data. 
20 The user, through a display in the 3.5 Configure Tour process, 
identifies what types of data is collected during his field data 
collection session. By identifying the types of data collected, 
the system applies the appropriate grammars and semantics to 
translate the data collected in the field into database records. 
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The user also enters his name^ organization and other relevant 
information. 

The user, through a series of displays in the 3.1 Merge 
Repository Updates process, merges the data collected in the field 
into the central Repository l.M. The assets, conditions, defects, 
and repairs are compared to the appropriate layer of historical 
data. Any discrepancies in the data are flagged and presented to 
the user for resolution. A discrepancy is identified when the new 
data is not consistent with the data already resident in the 
central Repository l.M. After discrepancies in the data are 
resolved, the user approves the changes and the central Repository 
l.M is updated. 

The 3-5 Collect DGPS Data function continuously collects GPS 
reference data from a connectable Reference GPS Receiver and stores 
it in central Repository l.M. This data is used to correct errors 
in the field collected GPS data. This correction can be performed 
post-processed or in real time. 

The Repository l.M data contains all the data for the system 
including all data stores discussed in earlier figures. This is 
data collected in the field, historical data, data used but not 
changed by the system, and reference data. Central Repository l.M 
contains, as a minimum, the following historical data: Assets, 
Conditions, Defects, and Repairs. Central Repository l.M contains, 
as a minimum, the following reference data: DGPS Data, Grammars, 
Semantics, and Imagery. 
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The Tour l.F data store contains the information collected in 
the field and statistics about the field session. The information 
contained in the tour is at a minimum: the inspector, data, 
duration, type of inspection, and correctly formatted repository 
updates. The 3.2 Extract Field Data process provides the function 
of combining tour data with other historical data stores for 
processing and use by the user. 

Figure 5 shows an example of data collection in the field. 
Figure 5 shows a vehicle V traveling north on Elm street. Figure 5 
shows the position of the vehicle V by its GPS points and shows two 
observation events indicated by the numerals 1 and 2, The data 
input from the observation events is voice data, indicated by the 
quotations in Figure 5. 

Figure 6 shows the processing sequence for data conversion for 
the two specific observation events 1 and 2 identified in Figure 5. 
Figure 6 also shows the semantic analysis of associating 
observation event 2 to observation event 1. The results of the 
semantic analyses are indicated by the inclined block arrow in the 
lower part of Figure 6. 

Figure 7 is the diagrammatic view of the four primary 
observations types. These four observations represent the possible 
data collected in the field and maintained in the Repository l.M 
and are described in more detail immediately below. 
Asset: 

Asset represents objects in the field that the user wishes to 
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track and maintain. Examples of assets are: street signs, side 
walks, and curbs. Assets can be related to other assets. For 
example, a street sign that has one post and two signs attached can 
be represented as three assets that are associated together. Both 
5 Assets and Defects (below) have a spatial type (e.g., point, linear 
or polygonal) . The spatial type and the associated layers 
information define how the asset information is correlated to other 
GIS layers during the automatic correlation processing shown in 
Figure 3. 

10 For example, street sign assets may be associated to a side 

GIS layer. This association defines that the location of the sign 
asset should be altered during processing to snap (or adjust) its 
position to be on the street edge, not in the street. Similarly, 
for defects, such as a concrete defect, (e.g., a crack), will be 

15 associated to the concrete network asset layer, which in turn is 
associated with the street edge layer. 
Condition 

Condition represents attributes of an asset that change over 
time and or position. The condition of the assets may be 
20 established in the system through grammar tables to allow the user 
to collect a predefined range and classes of conditions. For 
example, the conditions for street sign could be good, fair, and 
poor. 

Defect 

25 Defect represents a defined flaw in the asset that affect the 
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health or goodness of the asset. Defects can also be set through 
grammars to reflect a type of defect or a severity. 
Repair 

Repair is the removal of a defect. As a repair is made the 

5 central data Repository l.M can be updated to reflect the repair 
and the defect is then automatically removed from the database. 

The diagrammatic view of Figure 7 illustrates the four primary 
types of data maintained within central Repository l.M of the 
system shown in Figure 1 and also the possible relationships of the 

10 types of data. In Figure 7 (as illustrated by the diagram box in 
the bottom left hand corner of Figure 7) the arrows indicate the 
possible associations of the data structure relationships. Thus, 
as illustrated in Figure 1, Assets can always be associated with 
other Assets, Condition must be associated with an Asset, Defect 

15 must be associated with an Asset, and Repair can be associated only 
with a Defect. Figure 7 also shows the structure for each of the 
primary data types. The processing information portion of the 
structure of each primary observation type is embodied in the 
association (indicated by the arrows), the Spatial Type 

20 information, and the Storage Layer and Associated Layers 
information. Each of the primary observation types also has 
Location and Attributes in its structure. 

As noted above in the Brief Description of the Drawing Views, 
Figure 8 required too much illustration area to be capable of being 

25 shown on one sheet of drawings and was therefore composed of 



60063207_2DOC 



-25- 



Figure 8A (on one sheet of drawings) and Figure 8B (on the 
succeeding sheet of drawings) . Since it was necessary to show 
Figure 8 on two sheets, the textual content of Figure 8 is also set 
out below in this text for convenience in reference. 

Figure 8 is an example grammar of the type used in Figures 5 
and 6 but for a specific asphalt distress observation type. Each 
of the boxes shown in Figure 8 represent different sentence types. 
The two observation events illustrated in Figure 5 correspond to 
the respective top box and bottom box in Figure 8. The semantic 
information identifying that the second sentence is a modifier of 
the first sentence is illustrated by the two dash lines in 
Figure 8: the first dashed line going from ^^Tagrblob" up to the 
term "blob" and the second dashed line going from "Tag: area" up to 
"area" in the Observation Template. The observation statements in 
Figure 5 correspond to the Recognized Text 2. A in Figure 2, and the 
Reference Observation Semantics l.E of Figure 2 correspond to the 
information contained in the asphalt project grammar of Figure 8. 

As noted above. Figure 8 is an example grammar to the type 
used in Figures 5 and 6 but for a specific asphalt distress 
observation type. This example grammar illustrates one possible 
implementation of our method. There are two main sections 
illustrated in Figure 8: the Observation Templates and the 
Sentence Templates. Each of the spoken sentences and the resulting 
Observation Templates are shown for the examples used in Figures 5 
25 and 6 . 
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In the first Observation Template, shrparea, the structure of 
the resulting observation is defined by the specification enclosed 
by the 'Ml". The "%s" identifies the type of GIS record to create. 
The "%t" identifies that time is to be included. The "%p" 
5 identifies that location is to be included. The ''%e" identifies 
the several different slot values that are to be included (note the 
"".•ceriter" following the streetpos specification indicates that the 
value of center is a default) . The "%m" identifies that there is a 
future modifying statement to include, and if not found, then 
^..10 ^^hloh,sqft,50" is the default. The semantic relationship between 
the two separate verbal sentences is further illustrated by the 
dashed lines that indicate associations between templates, and 

!L" -ii 

I'' 7; i; 

'z\ between sentences and templates. 

Figure 8 further illustrates the semantic structure of the 
^il5 sentence templates. Each sentence, which corresponds to a set of 
LI possible verbal statements, is composed of slots. The information 
ill of how slot values are transferred to the observation record is 
defined by the PrcType attribute of each slot. 

For the first sentence ''shrpdistressarea'' each of the slots 
20 are copied into the resulting observation record based on slot tag. 
For the '"areasqft" sentence the numeric values are combined to form 
a true number that is, by convention, assigned to the ^'area" slot, 
with tag '"sqft," and that is then copied into the ''sqft%n" 
specification of the '"blob" Observation Template. In this case the 
25 ''%n" implies a numeric value required. The result of using this 
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semantic information enables the two distinct verbal observations 
made in the examples of Figures 5 and 6 to be combined 
automatically into one resulting GIS record. 

Figure 9 illustrates graphically the data correlation process 
for the examples illustrated in Figures 5, 6, and 8. 

While data collection is in progress, GPS data points are 
continuously collected, as well as the audio data and the other 
sensor data (see Figure 2). The GPS data record contains the 
location as well as the time-stamp for that location. 

When the system detects voice input by the user, a raw 
observation is created. This raw observation consists of the 
recorded voice and a time-stamp. Time is used as the 
synchronization key between all of the independent data streams: 
GPS, Audio, and Sensor. 

The GPS data points are then compressed into a series of 
tracks (vectors and arcs) that represent the route taken by the 
user. Each of the track records consist of a start and stop 
position. An observation record's location information is 
determined using time and the GPS data to interpolate the location 
and the associated track and position along the track. The record 
consists of the observations text data and other sensor data, the 
track it was associated to, and the position along the track that 
the observation was made. These pieces of information are used to 
correlate the route taken and the observations made to the 
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underlying network segments, which in this example are the street 
segments that were driven. 

In the example shown, the user drives the city streets and 
makes observations about the condition of the streets. A typical 

5 point observation statement is ^'hole medium." This observation is 
correlated to the street network, and a record is added to the 
Asphalt Distress Layer of the GIS. An example of a running 
observation is the combination ""^Segment Start", ^""Surface Asphalt" 
and "Segment End". These statements make a running observation 

10 which would be converted into a series of Asphalt Surface records 
for each street segment, and partial segment driven over between 
the ^'Segment Start" and '^Segment End" statements. 

Thus, as shown in Figure 9 the collected GPS data is converted 
into the Track l.F data. The Track l.F data is correlated with the 

15 Street Network data. Figure 9 also shows Defect data being loaded 
into its Asphalt Distress Layer. This Defect data from the Asphalt 
Distress Layer is then combined with the Street Network correlation 
results to create the association of the Defect with the Asset. 
The process from the GPS data layer to the track data layer 

20 (illustrated diagrammatically in Figure 9) is also illustrated by 
the 1.2 Vectorize Location Data process in Figure 2. The linkage 
from the track layer to the street network layer (illustrated in 
Figure 9) is also illustrated by the 2.1 Correlate Tracks To 
Network process in Figure 3. The input of the Defect data into the 

25 Asphalt Distress Layer (illustrated in Figure 9) is also 
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illustrated by the 1.6 Interpret Text process of Figure 2. The 
linkage between the Asphalt Distress Layer and the Street Network 
Layer (illustrated in Figure 9) is also illustrated by the 2.3 
Correlate Observation To Network process in Figure 3, Figure 9 

5 diagraimnatically illustrates the example of Figure 8 with respect 
to the two events noted on Elm Street as illustrated in Figure 5. 

While we have illustrated and described the preferred 
embodiments of our invention^ it is to be understood that these are 
capable of variation and modification^ and we therefore do not wish 

10 to be limited to the precise details set forth, but desire to avail 
ourselves of such changes and alterations as fall within the 
purview of the following claims. 
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